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EXECUTIVE SUMMARY 
Office of the Inspector General 
Training Evaluation and Management System (TEAMS) IT Audit, Phase I 
(Fiscal Year 2007/2008) 


PURPOSE 


Pursuant to Consent Decree Paragraph 137, the Office of the Inspector General (OIG) conducted 
its Training Evaluation and Management System (TEAMS) II Audit (Audit). Specifically, 
Paragraph 137 requires that the OIG, between 6-12 months following the implementation of 
TEAMS II and on a regular basis thereafter, audit the quality and timeliness of the Los Angeles 
Police Department’s (LAPD or Department) use of TEAMS II to perform the tasks identified in 
the protocol described in Consent Decree Paragraph 47. 


AUDIT METHODOLOGY 


The OIG assessed system-generated action items, the Department’s handling of disputed data 
corrections in key source systems, and employees’ access to their own TEAMS reports in this 
Phase I Audit. The six subparagraphs of Paragraph 47 covered were d, e, h, j, k (1), and 1. 
Additionally, the OIG assessed the Department’s performance in accordance with Special Order 
No. 20, 2006, Special Order No. 22, 2007, Chief of Police Notices, and other procedures which 
relate to TEAMS II usage. 


Generally, the OIG used a 95 percent confidence level, with an expected error rate of six percent 
and a plus precision of seven percent when selecting its random samples. In some cases, in 
addition to the original random sample, the OIG expanded test work by selecting a supplemental 
judgmental sample based on risks. For some of the test work, the OIG assessed the entire 
population to draw conclusions. 


The timeframe under audit was between January 31, 2007 (when the system-generated action 
items were first activated) and August 31, 2007. Due to constant fixes and patches being applied 
in TEAMS II, many of the significant exceptions identified in the beginning of the Audit were 
re-visited in August 2007. Unless otherwise stated, the exceptions included in this report are 
those which were not fixed by then. 


SUMMARY OF RESULTS 


e Risk Management Information System (RMIS) may be generating more action items 
(approximately 2,500 action items for 2,000 employees per year) than the Department can 
cost-effectively and timely complete. The Department has advised that the volume of 
system-generated action items is more than originally projected. One reason for so many 
action items is that action items are being generated for some employees that have had very 
limited activity related to the threshold, and these action items do not appear to be warranted. 
For example, in the OIG’s sample of 32 action items, five (16%) action items were generated 
due to an officer receiving one complaint or being involved in one use of force, but because 
they had a limited number of arrests for the period (ranging from one to six), the action item 
was generated. 
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e Per Special Order No. 20, 2006 and Special Order No. 22, 2007, Risk Management Group 
(RMG) was assigned TEAMS II audit responsibilities. As of October 10, 2007, they had not 
completed any TEAMS II audits. However, an audit plan is being prepared with consultation 
provided by the Audit Division. 


e For 14 of 23 action items in another sample, there was no evidence per the TEAMS II report 
that the required follow-up action was taken. As of August 22, 2007, it had been 105 to 147 
days, since the final dispositions were made, with a mean of 120 days. 


e For 19 of the 32 sampled action items, there was no indication that the supervisor met with 
the involved employee before the action item was completed, as required by an 
Intradepartmental Correspondence (15.2) issued on January 29, 2007, by Consent Decree 
Bureau. 


e As of August 22, 2007, 227 pending action items were past due (i.e., not completed within 30 
days), with 73 past due more than 60 days, ranging to 144 days. 


e For five of 13 requests sent from Use of Force Review Division (UOFRD) to TEAMS II 
Development Bureau (TEAMS II Bureau) for correcting historical disputed data in the Use 
of Force System, the correction was not made by TEAMS II Bureau as of August 20, 2007. 
It had been more than five months since the requests were made by UOFRD. 


e There was no mechanism designed within RMIS to force the investigating supervisors to 
open reports that are mandated to be reviewed. 


On October 10, 2007, the OIG met with the Commanding Officer of TEAMS I Bureau, an 
Officer-in-Charge in RMG, and other key personnel from both entities to discuss the Audit’s 
findings in detail. At that time, TEAMS II Bureau and RMG indicated agreement with most of 
the Audit’s findings and provided the OIG with action plans that have been incorporated into the 
report where appropriate. 


OFFICE OF THE INSPECTOR GENERAL 
FISCAL YEAR 2007/2008 
TRAINING EVALUATION AND MANAGEMENT SYSTEM (TEAMS) IT 
AUDIT, PHASE I 


PURPOSE 


Pursuant to Consent Decree Paragraph 137, the Office of the Inspector General (OIG) conducted 
its Training Evaluation and Management System (TEAMS) II Audit (Audit). Specifically, 
Paragraph 137 requires that the OIG, between 6-12 months following the implementation of 
TEAMS II and on a regular basis thereafter, audit the quality and timeliness of the Los Angeles 
Police Department’s (LAPD or Department) use of TEAMS II to perform the tasks identified in 
the protocol described in Consent Decree Paragraph 47. 


BACKGROUND 


The Consent Decree mandated the Department to enhance its legacy information system known 
as TEAMS to become TEAMS II, which provides the capability to perform complex employee 
and organizational performance analysis to identify at-risk employees and organizations. 
TEAMS II is composed of 14 independent personnel performance source applications/databases 
which contribute selected data into TEAMS II.’ The majority of these applications were existing 
applications in TEAMS but some were newly developed. The newly developed applications 
include the Deployment Period System (DPS) which involves on-line deployment/personnel 
status reporting, the Use of Force System (UOFS) which provides use of force data, the 
Complaint Management System (CMS) which provides personnel complaint data, and the Risk 
Management Information System (RMIS) which generates action items and provides various 
analytical reports. 


RMIS serves as a tool for supervisors and managers to conduct regular reviews of individual and 
organizational performance. When RMIS receives data indicating that an employee has engaged 
in a type of activity analyzed by RMIS, it performs a statistical analysis of the employee’s 
activity compared to that of the employee’s peer group.” The employee activity data analyzed by 
RMIS includes personnel complaints, use of force incidents, claims and lawsuits, preventable 
traffic collisions, pursuits, stops and arrests. Then, if the employee’s activity exceeds a particular 
threshold setting designed in RMIS, RMIS will automatically generate an action item. The 
action item is then forwarded to the employee’s supervisor for review. The investigating 
supervisor will document his/her review in an “investigative narrative” field and choose an 
appropriate disposition (e.g., complaint, training, no action, etc.). Upon completion, the 
investigating supervisor will forward the review of the action item up his/her chain of command 
to the Bureau commanding officer who will finish the review. 





The 14 applications are Canine System (K-9), Claims/Lawsuit Information System (CLIS), Complaint 
Management System (CMS), Comprehensive Automated Permit System (CAPERS), Consolidated Crime 
Analysis Database (CCAD), Department Employees Commendation System (DECS), Deployment Period System 
(DPS), Fleet Safety Reporting (FSR), Shooting Qualification and Bonus System (SQUABS), System to Organize 
Pedestrian/Vehicle Stops (STOPS), Traffic Information System (TIS), Training Management System (TMS), Use 
of Force System (UOFS), and Vehicle Pursuit System (VPS). 

? Every Department employee is assigned to a peer group based on the type of work he/she performs. There are a 
total of 33 peer groups in RMIS. 
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Action items were first activated on January 31, 2007 and completely “rolled-out” Department- 
wide as of March 12, 2007. As of July 31, 2007, there have been 1,232 action items generated 
involving 952 employees, with review finalized for 788 of these action items. 


SCOPE AND METHODOLOGY 


In addition to the previously mentioned system-generated action items (AI), RMIS provides the 
capability for supervisors to create their own action items, known as supervisor action items 
(SAI). In order to effectively and efficiently assess if the Department and TEAMS II are in 
compliance with Consent Decree Paragraph 47, the OIG decided to conduct this Audit in two 
phases. The OIG assessed system-generated action items, the Department’s handling of disputed 
data corrections in key source systems, and employees’ access to their own TEAMS II reports in 
this Phase I Audit. The six subparagraphs of Paragraph 47 covered were d, e, h, j, k (1), and 1. 
Additionally, the OIG assessed the Department’s performance in accordance with guidelines 
outlined in the RMIS User Guide; the Intradepartmental Correspondence (15.2) dated 

January 29, 2007, to all Commanding Officers from the Commanding Officer at Consent Decree 
Bureau; Special Order No. 20, 2006; Special Order No. 22, 2007; Chief of Police Notices; and 
other procedures which relate to TEAMS II usage. The OIG will assess supervisor action items 
in the Phase II portion of the Audit, covering subparagraphs a, b, c, f, g, 1, k (2), and m of 
Paragraph 47. 


Generally, the OIG used a 95 percent confidence level, with an expected error rate of six percent 
and a plus precision of seven percent when selecting its random samples. In some cases, in 
addition to the original random sample, the OIG expanded test work by selecting a supplemental 
judgmental sample based on risks. For some of the test work, the OIG assessed the entire 
population to draw conclusions. 


The timeframe under audit was between January 31, 2007 (when the system-generated action 
items were first activated) and August 31, 2007. Due to constant fixes and patches being applied 
in TEAMS II, many of the significant exceptions identified in the beginning of the Audit were 
re-visited in August 2007. Unless otherwise stated, the exceptions included in this report are 
those which were not fixed by then. 


SUMMARY OF RESULTS 


e Risk Management Information System (RMIS) may be generating more action items 
(approximately 2,500 action items for 2,000 employees per year) than the Department can 
cost-effectively and timely complete. The Department has advised that the volume of 
system-generated action items is more than originally projected. One reason for so many 
action items is that action items are being generated for some employees that have had very 





> During the review of system-generated action items and assessment of the quality of the supervisor's review and 
pursuant to Consent Decree Paragraph 138, the OIG examined and assessed the applicable officer's TEAMS II 
information. 


Training Evaluation and Management System (TEAMS) II Audit, Phase I 
Page 3 of 13 
1.0 


limited activity related to the threshold, and these action items do not appear to be warranted. 
For example, in the OIG’s sample of 32 action items, five (16%) action items were generated 
due to an officer receiving one complaint or being involved in one use of force, but because 
they had a limited number of arrests for the period (ranging from one to six), the action item 
was generated. To address this concern, the Department is planning to re-assess this area and 
then present any proposed modifications to the Department of Justice (DOJ) and the 
Independent Monitor (Monitor) in early 2008. 


e Per Special Order No. 20, 2006 and Special Order No. 22, 2007, Risk Management Group 
(RMG) was assigned TEAMS II audit responsibilities. As of October 10, 2007, they had not 
completed any TEAMS II audits. However, an audit plan is being prepared with consultation 
provided by the Audit Division. 


e For 14 of 23 action items in another sample, there was no evidence per the TEAMS II report 
that the required follow-up action was taken. As of August 22, 2007, it had been 105 to 147 
days, since the final dispositions were made, with a mean of 120 days. 


e For 19 of the 32 sampled action items, there was no indication that the supervisor met with 
the involved employee before the action item was completed, as required by an 
Intradepartmental Correspondence (15.2) issued on January 29, 2007, by Consent Decree 
Bureau. 


e As of August 22, 2007, 227 pending action items were past due (i.e., not completed within 30 
days), with 73 past due more than 60 days, ranging to 144 days. 


e For five of 13 requests sent from Use of Force Review Division (UOFRD) to TEAMS II 
Development Bureau (TEAMS II Bureau) for correcting historical disputed data in the 
UOFS, the correction was not made by TEAMS II Bureau as of August 20, 2007. It had 
been more than five months since the requests were made by UOFRD. 


e There was no mechanism designed within RMIS to force the investigating supervisors to 
open reports that are mandated to be reviewed. 


On October 10, 2007, the OIG met with the Commanding Officer of TEAMS IJ Bureau, an 
Officer-in-Charge in RMG, and other key personnel from both entities to discuss the Audit’s 
findings in detail. At that time, TEAMS II Bureau and RMG indicated agreement with most of 
the Audit’s findings and provided the OIG with action plans that have been incorporated into the 
report where appropriate. 
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DETAILED FINDINGS 


Subparagraph 47d_- Performance Threshold Formulas 


Subparagraph 47d requires the establishment of guidelines for numbers and types of incidents to 
cause the activation of an action item in TEAMS II, and the frequency of these reviews. 


In order to assess whether the Department met the mandate of this subparagraph, the OIG 
obtained the RMIS formulas for all 15 performance thresholds, as agreed upon by the City and 
the Department of Justice for activating a system-generated action item. The OIG also obtained 
RMIS readiness testing materials utilized by the contractor Sierra Systems Group Inc. (Sierra) 
and reviewed by TEAMS II Bureau, and other relevant design documents. Additionally, the 
OIG obtained the daily final completed action items reports and pending action items reports. 
The OIG did not validate the accuracy of the 15 performance threshold formulas, which would 
have included detailed, technical testing. 


Assessment of Performance Thresholds 


Based on our review, the OIG noted that RMIS may be generating more action items 
(approximately 2,500 action items for 2,000 employees per year’) than the Department can cost- 
effectively and timely complete. The Department has advised that the volume of system- 
generated action items is more than originally projected. One reason for so many action items is 
that action items are being generated for some employees that have had very limited activity 
related to the threshold, and these action items do not appear to be warranted. For example, in 
the OIG’s sample of 32 action items, five (16%) action items were generated due to an officer 
receiving one complaint or being involved in one use of force, but because they had a limited 
number of arrests for the period (ranging from one to six), the action item was generated.” 


Department Action Plan 
The Department is planning to re-assess the peer group Settings, performance thresholds, 


and disposition types at the end of 2007 and submit its proposed modifications to the 
DOJ and the Monitor in early 2008. 





* As of August 22, 2007, there were 939 action items completed through final Bureau level review, of which 797 
received a disposition of “no action.” The “no action” disposition accounted for approximately 85 percent of the 
total action items. 

` The performance thresholds generating the five action items were No. 2 — Use of Force to Arrests (three action 
items) and No. 6 - Complaints to Arrests (two action items). 
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Subparagraph 47e _- Follow-up on Supervisory Actions Taken 


Subparagraph 47e requires guidelines be provided for appropriate follow-up actions to be taken 
based on reviews of information in TEAMS IL. 


In order to assess this subparagraph, the OIG obtained the Department policies, procedures, and 
the Final Completed Action Items Report as of May 11, 2007. 


Assessment of Follow-up Actions Taken as a Result of Action Item Disposition 


Among the total of 293 final completed action items as of May 11, 2007, 247 (85%) action items 
received a “no action” disposition, and 46 (15%) action items received a disposition other than 
"no action," requiring some type of follow-up action. From these 46, the OIG selected a random 
sample of 23. For 14 of the 23 (61%) sampled action items, there was no evidence per the 
TEAMS II report that the mandated follow-up action was taken as of August 22, 2007. It had 
been 105 to 147 days since the final dispositions were made, with a mean of 120 days. The 
following table provides details on the dispositions and a breakdown of the follow-up action 
status for these 23 action items. 


Breakdown of Follow-up Actions for the 23 Sampled Action Items 























Total Total No. of AIs With Total No. of AIs Without 
Disposition No. of | Evidence per TEAMS II Evidence per TEAMS II 
Sampled | Report that Follow-up Report that Follow-up 
Als Action Was Done Action Was Done 

Commendation 6 2 4 
Comment Card 6 2 4° 
Informal Meeting 6 2” 4 
Training 3 1 2 
Complaint 1 i 0 
RMEC Referral 1 1° 0 
Total 23 9 14 




















In our sample of 23 action items, there were six action items with an “informal meeting” 
disposition, but only two of them documented in the “investigative narrative” that the informal 
meeting was conducted by the supervisor. Special Order No. 22, 2007 states that an informal 
meeting shall be “used when the supervisor meets with the employee and conducts an informal 
counseling session.” However, the OIG found that the Department did not provide any 
guidelines for documenting that an informal meeting was conducted. Given that the “informal 
meeting” disposition accounted for 37 percent of all the action items which received a final 
disposition other than “no action” (as of August 22, 2007), the OIG believes that the Department 





° The OIG requested the Divisions to locate these comment cards and none of them were located. 

7 These informal meetings were documented in the investigative narratives. 

$ The complaint was for “failure to qualify” which was initiated prior to the generation of this action item. 

? This employee had already been referred to the Risk Management Executive Committee (RMEC) prior to the 
generation of this action item and is currently being monitored. 
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should provide guidelines for documenting the extent of the counseling and the date it was 
conducted.” 


Additionally, based on our review of Special Order No. 20, 2006 and Special Order No. 22, 
2007, the OIG noted that the responsibility to audit TEAMS II is assigned to RMG. As of 
October 10, 2007, RMG had not completed any such audits due to a lack of auditing resources. 


Department Action Plan 


TEAMS II Bureau and RMG concurred with the OIG’s finding and provided us with a few 
action plans, as follows: 


e TEAM II Bureau and RMG will include the re-assessment of the disposition types in the 
year-end evaluation of peer group settings and performance threshold calculations and 
then submit any proposed changes to the DOJ and the Monitor. 


e RMG will follow-up on the findings identified in this subparagraph with each related 
entity and ensure that the follow-up action is taken. Prior to the audit’s conclusion, two 
“commendation” dispositions and two “training” dispositions were updated to the 
related four officers’ TEAMS II reports. 


e RMG will include the assessment of follow-up action taken (if applicable) in their 
upcoming audit. 


e RMG, in consultation with Audit Division, is preparing an audit plan for their first 
TEAMS II audit, which will primarily assess compliance with Special Order No. 20, 2006 
and related policies and procedures. 


Subparagraph 47h — Opening of Mandated Reports 


Subparagraph 47h requires that actions taken as a result of information from TEAMS II be based 
on all relevant and appropriate information, and not solely on the number of percentages of 
incidents in any category recorded in TEAMS I. 


Assessment of the Supervisor Opening of Mandated Reports for Als 


Special Order No. 22, 2007, entitled “Duty to Conduct and Document Individual Performance 
Assessments” dated June 4, 2007, was distributed Department-wide the week of July 16, 2007. 
Special Order No. 22, 2007, listed multiple mandated reports to review for three different levels 
of involved employees. In order to assess this subparagraph, the OIG researched whether the 
investigating supervisor (the first level reviewer of the action item) who is the involved 





10 As of August 22, 2007, there were 939 action items completed through final Bureau level review, of which, 142 
received a disposition other than “no action.” The “informal meeting” disposition accounted for 52 of the 142 
(37%) action items. 
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employee’s direct supervisor could conduct the minimum review as required by Special Order 
No. 22, 2007. The OIG noted in RMIS that mandated reports are made available to the 
supervisor via a hyperlink; however, RMIS does not force the supervisor to open any of these 
reports to evidence their review. In order to assess the Department’s compliance with Special 
Order No. 22, 2007, the OIG selected a random sample of 15 action items from all the action 
items which were created and completed in August, 2007. The OIG’s detailed review of the 
RMIS monitoring report, which tracks the history of opening each report by the investigating 
supervisor, disclosed that for 8 of the 15 (53%) sampled action items, it appears that the 
investigating supervisor did not open all the mandated reports. Refer to the table below for 
breakdown of supervisor opening of mandated reports. 


Breakdown of Supervisor Opening of Mandated Reports 
































Total No. of | Total No. of | Percentage 
Report Name Applicable | Reports Not of Non- 
Reports Opened Compliance 
TEAMS II Report (RMIS-01 or A01) 15 4 27% 
Summary of Employee Activity (RMIS-02 or A02) 15 5 33% 
Comparison of Employee Activity to Selected 8 7 88% 
Organizations (RMIS-03 or A03)"! 
Threshold Activation Report (RMIS-10 or R10) 15 5 33% 





Assessment of the Design of RMIS Check Boxes and Linked Reports for SAIs 


The OIG also noted that there are similar system design deficiencies for Supervisor Action 
Items. Specifically, there are three check boxes that a supervisor must check off to evidence 
his/her review, as follows: 1) Have you examined other pending complaints, if any? 2) [Have 
you] examined the TEAMS II report in detail? 3) [Have you] examined the Summary of 
Employee Activity and other linked reports above? Based on our general review of the system 
design in RMIS, the OIG found that despite that RMIS forces the reviewers to check off the 
aforementioned three boxes evidencing that they have examined all the mandated reports which 
were made available via a hyperlink before going to the next screen, RMIS does not force the 
reviewers to actually open any of these reports. The OIG is concerned that, in the current design, 
reviewers could simply check off all the boxes indicating that they have reviewed the mandated 
reports without ever opening any of the reports. 


Furthermore, for both system-generated action items and supervisor action items, optional 
reports and mandated reports are being listed together without any differentiation, making it 
unclear to the reviewers which reports are mandated versus optional to review. 


The OIG believes that without fixing the deficiencies in RMIS, the supervisors may not open and 
review the mandated reports as required by Special Order No. 22, 2007. 





l Per Special Order No. 22, 2007, for supervisor and manager levels of employees being reviewed, Comparison of 
Employee Average Activity for Selected Organizations (E06) is one of the mandated reports to review; however, 
it was replaced by an optional report A03. When the OIG compared reports A03 and E06, it was noted that A03 
contained all the information in E06. Thus, there is no risk associated with reviewing A03 instead of E06. 
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Department Action Plan 


TEAMS II Bureau concurred with the OIG’s finding and advised the OIG that they will 
request Sierra to “patch” RMIS, so as to differentiate the mandated reports from the 
optional reports and force the reviewers to open the mandated reports before they complete 
the action item. 


Subparagraph 47j (1) - Employee’s Access to Their TEAMS II Report 


Subparagraph 47; requires each LAPD officer be able to review, on a regular basis, all 
personally-identifiable data about them on the TEAMS II report in order to ensure the accuracy 
of that data. 


Assessment of Employees’ Access to Their TEAMS II Report 


In order to assess this subparagraph, the OIG judgmentally selected a sample of 33 officers with 
one officer from each geographical area, each division of Special Operations Bureau, and 
Detectives Bureau. In addition, the OIG obtained the complete listing of all employees with 
access to RMIS to view their personal TEAMS II reports. The OIG then reviewed the sampled 
officers’ access roles in the RMIS user access listing and found that all 33 officers had access to 
view their TEAMS II reports. 


Additionally, the OIG assessed all 25 sworn and civilian employees who were identified with no 
peer groups to determine whether they could view their own TEAMS II reports. This follow-up 
review disclosed that one reserve officer could not access his TEAMS II report.'” However, this 
appears to have been an isolated incident, possibly due to the reserve officer not having a work 
assignment input into DPS. 


Department Action Plan 


TEAMS II Bureau concurred with the OIG’s finding and rectified this employee’s access 
which allowed him to view his TEAMS II report. 


Subparagraph 47j (2) — Disputed Data Correction 


Subparagraph 47) further requires that a protocol be adhered to for correcting data errors 
discovered by officers in their review of the TEAMS II data. 


Per Special Order No. 20, 2006, after receiving a request to correct inaccurate or incomplete 
data, the commanding officer of the entity responsible for correcting data in TEAMS II shall: 1) 
investigate the disputed information; 2) ensure that a resolution is obtained within 30 days of the 





12 This reserve officer was assigned to the Area in May 2007. 
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request; 3) correct the disputed information, if appropriate; and, 4) notify the employee's 
commanding officer of the resolution in an Intradepartmental Correspondence, Form 15.2 (15.2). 


Based upon our risk assessment, the OIG decided to focus on the major source systems which 
feed data into RMIS. As such, the OIG selected UOFS and CMS. The OIG assessed the 
procedures within each entity responsible for the data entry/maintenance of the aforementioned 
source systems. 


Assessment of CMS Disputed Data Corrections 


A total of 66 entries for CMS data correction were identified in the CMS disputed data correction 
log created by the Review & Evaluation Section (R&E) of Internal Affairs Group. A random 
sample of 22 recent entries involving 18 employees was selected. 


Based on our review, the OIG found that R&E made all the necessary corrections of disputed 
data in CMS. However, of the 22 data correction requests, only five were handled in strict 
accordance with Department policies, the remaining 17'° were not, as follows: 


e For four of the disputed data correction requests, a request was not made via a 15.2; 

e For six of the disputed data correction requests, a resolution was not obtained within 30 days 
after receiving the Area’s request, ranging from two to 12 days late; and, 

e For 13 of the disputed data change requests, a notification of resolution was sent via email or 
telephone instead of a 15.2 to the employee’s commanding officer. 


As a result of our assessment, R&E formalized the internal procedure to re-emphasize, to their 
designated personnel, the procedures for handling disputed data correction in CMS. 


Assessment of UOFS Disputed Data Corrections 


The UOFRD provided the OIG with all 13 UOF disputed data correction requests involving 12 
officers. Based on our testwork results, for all 13 UOF cases, the OIG noted that a 15.2 was sent 
by UOFRD to TEAMS II Bureau requesting them to make the data correction. UOFRD could 
not make these corrections since the applicable cases were historical cases (cases which were 
uploaded to UOFS via data conversion). 


Based on our review, the OIG found that none of these 13 requests were handled in strict 
accordance with Department policies, as follows: 


e One request was done via email by the involved officer, not via a 15.2; 

e For six of the requests, a resolution was not obtained within 30 days after the area submitted 
a 15.2. For three of the aforementioned six requests, there was no evidence that a 15.2 was 
sent to the Area; and, 





13 Some of the data correction requests had more than one exception type. 
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e For ten of the requests sent from UOFRD to TEAMS II Bureau, the corrections had not been 
made as of May 30, 2007; 41 to 167 days after these requests were made by UOFRD. 
During the OIG’s revisit of these issues on August 20, 2007, the OIG noted that five of the 
ten disputed data corrections were made. 


Department Action Plan 


TEAMS II Bureau concurred with the OIG’s finding and as of the audit report date, the five 
remaining disputed data corrections were made. Additionally, the Office of the Chief of 
Police Notice dated September 28, 2007, to all Department Personnel from the Commanding 
Officer at Consent Decree Bureau provided two templates (available on the TEAMS IT 
website) to standardize data correction in accordance with the mandates of the Consent 
Decree Paragraph 47(j) and Department policies. 


Subparagraph 47k (1) — Adequacy of Documenting Action Item Reviews 


Subparagraph 47k requires regular review by appropriate managers of all relevant TEAMS II 
information (1) to evaluate officer performance citywide, and (2) to evaluate and make 
appropriate comparisons regarding the performance of all LAPD units in order to identify any 
patterns or series of incidents that may indicate at-risk behavior. These evaluations shall include 
evaluating the performance over time of individual units, and comparing the performance of 
units with similar responsibilities. 


In our Phase I Audit, the OIG only assessed objective (1). Since TEAMS II Bureau is still 
designing a few reports for comparison of the performance of all LAPD units to satisfy 
objective (2), that objective will be assessed in the Phase II portion of the Audit. 


To assess objective (1), the OIG obtained the Final Completed Action Items Report as of May 
11, 2007, which contained a total of 293 final completed action items. The OIG selected a 
random sample of 32 action items from the six operational Bureaus (i.e., West Bureau, Central 
Bureau, Valley Bureau, South Bureau, Special Operations Bureau, and Detective Bureau). 
Additionally, the OIG obtained a report containing supervisor investigative narratives and 
information on which supervisors/commanding officers were involved in the review. 


Assessment of Documentation of Action Item Reviews 


The OIG assessed whether the documentation of the review of the action item and related 
rationale for the disposition in each investigative narrative appeared adequate. The OIG found 
that for 29 of the 32 (90%) sampled action items, the supervisor adequately documented the 
review of the action item and the related rationale for the disposition in the investigative 
narrative. There were three supervisors who appeared not to adequately document the extent of 
their reviews in the investigative narratives. When the OIG extended its testwork by evaluating 
all action items reviewed by these three supervisors, it was noted that two of the supervisors 
adequately documented their assessment of all other action items; but one supervisor exhibited 
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the same concern in another action item’s review. The OIG believes, in general, the Department 
adequately documented the reviews of the action items and related rationales for the dispositions 
in the investigative narratives based on the guidance provided to supervisors during Department- 
wide trainings on how to complete action items. 


Assessment of Supervisors’ Meeting with the Employees 


The 15.2 dated January 29, 2007, from the Commanding Officer of Consent Decree Bureau 
requires that, prior to completing the review, the investigating supervisor meet with the involved 
employee to discuss the action item and document the date and time of the meeting." Based on 
our review, the OIG noted that 19 of the 32 (59%) sampled action items did not indicate that a 
supervisor met with the involved employee before the action item was completed. Among the 13 
in which the supervisor documented that they met and discussed the action item with the 
involved employee, four did not indicate the date of the meeting. 


Department Action Plan 


RMG personnel plan to provide, in the RMIS User Guide, an updated narrative exemplar 
regarding how supervisors should document their meetings with employees. Also, RMG will 
cover this area in their upcoming audit. 


Subparagraph 471 - Timely Completion of Action Item Reviews 


Subparagraph 471 requires that the protocol shall provide for the routine and timely 
documentation in TEAMS II of actions taken as a result of reviews of TEAMS II information. 
Additionally, Department policy requires that an action item be completed within 30 days after it 
is generated unless an extension is approved by the commanding officer. 


Assessment of Timely Completion of Action Item Review 


To assess the timeliness of completion of action items Department-wide, the OIG obtained the 
Pending Action Items Report as of August 22, 2007. This report contained 502 pending action 
items, with 227 past due. Note per the aging analysis below that 73 of these 227 (32%) action 
items were past due more than 60 days, ranging to 144 days. 


Aging Analysis of 227 Past Due Action Items 
Past Due Days | Less than 30 Days | 31-60 Days | 61-90 Days | Over 90 Days 
Quantity 70 84 50 23 
Percentage 31% 37% 22% 10% 
































Regarding our original sample of 32 action items, 11 of them (34%) were not completed within 
the 30-day deadline, ranging from two to 15 days past due. 





14 In this Audit, the OIG only assessed whether the supervisors documented the date of the meeting with the 
employee. Documentation of the time of the meeting was not assessed. 
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The OIG noted that TEAMS II Bureau, RMG, and Sierra have held weekly action item meetings 
to monitor the progress of past due action items and have provided assistance to the reviewers to 
ensure timely completion. Moreover, the Office of Chief of Police Notice dated July 18, 2007, 
re-iterated that all action items shall be completed through Bureau level review within 30 days of 
being generated. As previously mentioned, the current RMIS thresholds are generating more 
action items than originally anticipated. The OIG believes that once the thresholds are modified, 
there should be a decrease in overall amount of action items, and as a result, the timeliness of 
completing action items might be improved. 


Department Action Plan 


In addition to the aforementioned on-going efforts, RMG is planning to include the 
assessment of timely completion of the action items review in their upcoming audit. 


Assessment of Action Items Being Recorded Accurately on Officers’ TEAMS II Reports 


A system-generated action item will appear on an employee’s TEAMS II report as “pending” 
after the investigating supervisor completes his/her first level of review. The “pending” status 
will remain until it has been reviewed by the employee’s Bureau. Upon final review and 
approval, it will change to indicate the final disposition of the action item. 


For the 32 sampled final completed action items, the final dispositions were accurately recorded 
on the related officers’ TEAMS II reports. 


OTHER RELATED MATTER 
Local Area Network Security 


The OIG has previously identified concerns with Local Area Network (LAN) security which, 
because of their confidential nature, are not discussed within this Audit Report. However, these 
concerns were discussed with Information Technology Division management and they have 
advised that these concerns will be adequately addressed in the near future through the 
implementation of new policies and procedures. 


RECOMMENDATIONS 
Based on our review, the OIG recommends the following: 


1) The Department should re-assess the peer group settings, performance threshold formula, 
and disposition types related to action item reviews, and make the necessary modifications. 


2) The Department should provide guidelines for the follow-up action due date and the extent 
of the documentation required for those disposition types that are not covered in the existing 
policies or procedures (e.g., informal meeting). 
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3) The Department should finalize its audit plan and commence TEAMS II audits, as required 
by the Special Orders. These audits shall assess areas of concern identified in this Audit, 
including but not limited to the following: 

a. Handling disputed data corrections, to ensure that they are done accurately and timely, as 
required by Special Order No. 20, 2006; 

b. Follow-up actions taken based on the review of system-generated action items and the 
subsequent dispositions; 

c. Supervisory documentation of meetings with employees to discuss action items prior to 
their completion; and, 

d. Timeliness of completion of action item reviews. 


4) The Department should “Patch” RMIS so as to differentiate the mandated reports from the 
optional reports and force the reviewers to open the mandated reports before they complete 
the action item. 


CONCLUSION 


Regarding system-generated action items, the OIG believes, in general, the Department 
adequately documented the reviews and the related rationales for the dispositions in the 
investigative narratives. However, the OIG is concerned that: 1) the current peer group and 
performance threshold settings in RMIS may be generating more action items than the 
Department can cost-effectively and timely complete; 2) approximately 32 percent of all 227 
past due action items were past due more than 60 days; 3) some action items lacked evidence 
that the required follow-up actions were being done; and, 4) RMIS does not have a mechanism to 
force the action item reviewers to open the mandated reports; and as a result, some investigating 
supervisors were not opening the mandated reports. All employees tested had access to their 
TEAMS II report, except for one isolated incident. Relative to CMS, disputed data correction 
requests were completely processed with the necessary corrections made, despite some 
procedural exceptions. However, relative to UOFS, disputed data correction requests sent from 
UOFRD to TEAMS II Bureau were not always completely processed with the necessary 
corrections made. 


Despite some concerns identified in this Audit, the OIG would like to recognize the multiple 
on-going efforts of TEAMS II Bureau and RMG to ensure continuous improvement in the 
effectiveness and user friendliness of TEAMS II. These efforts include, but are not limited to: 

1) weekly action item meetings to monitor the progress of past due items; 2) bi-monthly 
TEAMS II coordinator meetings to identify issues, and promote consistency and standardization; 
3) an on-line suggestion box to collect suggestions from users; and, 4) sending help desk staff 
out to the field to help users. 


The OIG believes with the implementation of the aforementioned action plans, progress shall be 
made. 


